Microscopic examination device and navigation method

ABSTRACT

A microscopic examination device includes: a camera that images an observation area of an examination subject observed with a microscope device so as to acquire an image of the observation area; and one or more processors including hardware, the one or more processors being configured to: align the image of the observation area with an area in a reference image, the area corresponding to the image of the observation area; generate a navigation map from the reference image by recording, on the reference image, a position of the image of the observation area in the reference image; calculate a direction of movement to an unobserved area in the navigation map, the unobserved area being an area where the position of the image of the observation area is not recorded; and present an access method to the unobserved area on a basis of the direction of movement.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of International Application PCT/JP2019/049050,with an international filing date of Dec. 13, 2019, which is herebyincorporated by reference herein in its entirety.

TECHNICAL FIELD

The present invention relates to a microscopic examination device and anavigation method.

BACKGROUND ART

In processes for manufacturing industrial products such as electronicparts, automobiles, and medical devices, abnormalities such asscratches, defects, impurities, and contaminants are examined visually.As technical innovation accelerates and customer needs shift everfaster, the lifecycle of products has shortened, and the demand forhigh-variety, small-lot production has grown. Visual examinationconducted with a microscope manually operated by a highly skilledengineer having high sensing performance is advantageous compared toautomated optical examination devices from the viewpoints of cost,flexibility, quality assurance, etc.

In visual examination, the operator manually moves the examinationsubject or the stage of a microscope to move the observation area of theexamination subject and to thereby observe all parts of the examinationsubject. Typically, the operator moves the observation area by using aspecific site within the examination subject as a marker to confirm theapproximate position of the observation area. However, when theobservation area is manually moved, it is difficult to unfailinglyobserve all parts of the examination subject. Upon identifying thepresence of an unobserved area, the operator switches the objective lensfrom a high-magnification lens to a low-magnification lens, confirms theposition of the unobserved area in a wide visual field, moves theunobserved area to the center of the visual field, switches theobjective lens back to the high-magnification lens, and observes theunobserved area. As such, visual examination that involves manuallymoving the observation subject takes effort and time. Moreover, sincethe observation area is very narrow, the operator may lose track of thepositions of already observed areas and the positions of unobservedareas, and may have difficulty determining whether or not all parts ofthe examination subject have been observed.

Meanwhile, there is a known microscope system for multiwell plateobservation, in which a current visual field range of the microscope isoverlaid on a navigation image that includes one entire well (forexample, see PTL 1). The operator can keep the track of the currentobservation position on the basis of the position of the visual fieldrange on the navigation image.

CITATION LIST Patent Literature

{PTL 1}

-   Japanese Unexamined Patent Application, Publication No. 2015-82099

SUMMARY OF INVENTION

An aspect of the present invention is directed to a microscopicexamination device comprising: a camera that images an observation areaof an examination subject observed with a microscope device so as toacquire an image of the observation area; and one or more processorscomprising hardware, the one or more processors being configured to:align the image of the observation area with an area in a referenceimage, the area corresponding to the image of the observation area, thereference image including an entirety of the examination subject;generate a navigation map from the reference image by recording, on thereference image, a position of the image of the observation area in thereference image; calculate a direction of movement to an unobserved areain the navigation map, the unobserved area being an area where theposition of the image of the observation area is not recorded; andpresent an access method to the unobserved area on a basis of thedirection of movement.

Another aspect of the present invention is directed to a navigationmethod for guiding an observation area of an examination subject to beobserved with a microscope device, the navigation method including:imaging the observation area to acquire an image of the observationarea; aligning the image of the observation area with an area in areference image, the area corresponding to the image of the observationarea, the reference image including an entirety of the examinationsubject; generating a navigation map from the reference image byrecording, on the reference image, a position of the image of theobservation area in the reference image; calculating a direction ofmovement to an unobserved area in the navigation map, the unobservedarea being an area where the position of the image of the observationarea is not recorded; and presenting an access method to the unobservedarea on the basis of the direction of movement.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram of the overall structure of a microscopicexamination device and a microscopic examination system according to afirst embodiment of the present invention.

FIG. 2 FIG. 2 is a diagram illustrating one example of a microscopedevice illustrated in FIG. 1 .

FIG. 3A illustrates an example of a reference image for a navigationmap.

FIG. 3B is a diagram illustrating one example of a navigation mapgenerated from the reference image illustrated in FIG. 3A.

FIG. 4A is a diagram illustrating one example of the display of anaccess method to an unobserved area on the navigation map.

FIG. 4B is a diagram illustrating another example of the display of theaccess method to the unobserved area on the navigation map.

FIG. 5A is a flowchart illustrating the operation of the microscopicexamination device illustrated in FIG. 1 .

FIG. 5B is a flowchart illustrating a process of aligning an observationimage with a reference image illustrated in FIG. 5A.

FIG. 5C is a flowchart illustrating a process of generating a navigationmap illustrated in FIG. 5A.

FIG. 6 is a diagram of the overall structure of a modification of themicroscopic examination device and the microscopic examination systemillustrated in FIG. 1 .

FIG. 7A is a flowchart illustrating the operation of the microscopicexamination device illustrated in FIG. 6 .

FIG. 7B is a flowchart illustrating a process of determining the accesssequence to unobserved areas illustrated in FIG. 7A.

FIG. 8 FIG. 8 is a diagram illustrating one example of a method fordetermining the access sequence of unobserved areas.

FIG. 9 FIG. 9 is a flowchart illustrating the operation of a microscopicexamination device according to a second embodiment of the presentinvention.

FIG. 10A is a diagram illustrating one example of the indicator of anaccess speed.

FIG. 10B illustrates one example of the indicator of the access speed.

FIG. 11A is a diagram illustrating another example of the indicator ofthe access speed.

FIG. 11B is a diagram illustrating yet another example of the display ofthe access speed.

FIG. 12 FIG. 12 is a diagram of the overall structure of a microscopicexamination device and a microscopic examination system according to athird embodiment of the present invention.

FIG. 13A is a flowchart illustrating the operation of the microscopicexamination device illustrated in FIG. 12 .

FIG. 13B is a flowchart illustrating a process of aligning anobservation image with a reference image illustrated in FIG. 13A.

DESCRIPTION OF EMBODIMENTS First Embodiment

A microscopic examination device 1 according to a first embodiment ofthe present invention will now be described with reference to thedrawings.

FIG. 1 is a diagram illustrating the overall structure of a microscopicexamination system 100 that includes a microscopic examination device 1.The microscopic examination system 100 also includes a microscope device20 and a display device 30, and the microscopic examination device 1 isconnected to the microscope device 20 and the display device 30.

The microscope device 20 is an optical microscope device of a type inwhich the operator manually moves the observation area of an examinationsubject S to be observed through an objective lens, and is used invisual examination of the examination subject S. The examination subjectS is, for example, an electronic component such as a circuit board, or acomponent part of an industrial product such as an automobile, anairplane, or medical equipment. The microscopic examination device 1generates a navigation map (see FIGS. 3B to 4B) for guiding theobservation area to an unobserved area of the examination subject S inthe visual examination using the microscope device 20, and displays thenavigation map on the display device 30.

FIG. 2 illustrates one example of the microscope device 20. Themicroscope device 20 illustrated in FIG. 2 is equipped with a stage 21on which the examination subject S is placed, an objective lens 22 forobserving the examination subject S on the stage 21, handles 23 a and 23b used by the operator to manually maneuver the stage 21, and an ocularlens 24. The handle 23 a is used to move the stage 21 in the X directionand the Y direction orthogonal to the optical axis of the objective lens22, and the handle 23 b is used to move the stage 21 in the Z directionthat runs along the optical axis of the objective lens 22. Through theocular lens 24, the operator can observe an optical image of theobservation area of the examination subject S formed by the objectivelens 22.

The microscope device 20 may be a stereoscopic microscope device, inwhich the examination subject S is held with the hands of the operatorand moved.

The display device 30 is a display device of any type, such as a liquidcrystal display, placed outside the microscope device 20. Alternatively,the display device 30 may be a display placed within the visual field ofthe ocular lens 24, or a display connected to the microscope device 20and designed to project an image onto the visual field of the ocularlens 24.

The digital image of the visual field of the ocular lens 24 may bedisplayed on the display device 30 outside the microscope device 20. Insuch a case, the operator may observe the digital image displayed on thedisplay device 30 instead of observing the optical image of theexamination subject S through the ocular lens 24. The display device 30may display the digital image and the navigation map side-by-side sothat the operator can easily compare the digital image and thenavigation map.

As illustrated in FIG. 1 , the microscopic examination device 1 includesan imaging unit 2, an imaging control unit 3, a memory unit 4, areference image input unit 5, a processor unit 6 that includes analignment unit 7, a navigation map generating unit 8, and an accesscalculation unit 9, and a navigation presenting unit 10.

The imaging unit 2 is a digital camera equipped with an imaging elementand a memory, and is connected to a camera port of the microscope device20. The imaging unit 2 captures an optical image of the observation areaof the examination subject S formed by the objective lens 22, andacquires an observation image, which is a digital image of theobservation area. The observation image is input from the imaging unit 2to the processor unit 6.

On the basis of the movement of the observation area and the imagingconditions, the imaging control unit 3 commands the imaging unit 2 toacquire a live image or a still image, which is an observation image.For example, on the basis of the live image acquired through the imagingunit 2, the imaging control unit 3 detects whether the observation areais moving or not, and commands the imaging unit 2 to acquire a stillimage under imaging conditions preset by the operator at a timing whenthe observation area has stopped moving. Alternatively, the imagingcontrol unit 3 may command the imaging unit 2, which is capturing liveimages, to acquire still images at particular time intervals.Acquisition of a still image at a timing when the observation area hasstopped moving and acquisition of still images at particular timeintervals may both be performed. Alternatively, in order to preventacquisition of multiple still images of the same visual field and toefficiently acquire still images, the imaging control unit 3 may detectswitching of the visual field on the basis of the live image and maycommand acquisition of a still image upon switching of the visual field.

The memory unit 4 has a reference image of the examination subject S anda program preliminarily stored therein.

The reference image is an image that includes the entirety of theexamination subject S. For example, the reference image is a designdrawing of the examination subject S, and the memory unit 4 stores theCAD data of the design drawing.

The program is a microscopic examination program that allows a processorto execute the processes that involve the imaging control unit 3, thereference image input unit 5, the processor unit 6, and the navigationpresenting unit 10. In other words, the functions of the imaging controlunit 3, the reference image input unit 5, the processor unit 6, and thenavigation presenting unit 10 described below are realized by theprocessor.

The reference image input unit 5 acquires the reference image of theexamination subject S from the memory unit 4, and inputs the referenceimage to the processor unit 6. The reference image input unit 5 mayacquire the reference image by a different method. For example, thereference image input unit 5 may acquire a reference image, which hasbeen obtained by any imaging device before starting the examination,from a device outside the microscopic examination device 1.

The processor unit 6 generates a navigation map by using the observationimage from the imaging unit 2 and the reference image from the referenceimage input unit 5, and calculates the information needed to reach theunobserved area on the basis of the navigation map.

The alignment unit 7 aligns the observation image with an area in thereference image that corresponds to the observation image. An existingtechnology is employed for alignment.

For example, the alignment unit 7 receives the reference image from thereference image input unit 5. The alignment unit 7 also reads theobservation image from the imaging unit 2, and generates, from theobservation image, a design image that serves as an observation imagefor alignment. The design image is a diagram corresponding to the designdrawing of the observation area of the observation image, and isgenerated from the observation image by using a learning model generatedby mechanical learning. Next, the alignment unit 7 calculates thesimilarity between the design image and local areas of the referenceimage, and the area with the highest similarity in the reference imageis aligned with the design image.

In one example, a preliminarily constructed learned network is used togenerate the design image. For example, a pair including an observationimage and a design image is learned by using an image generatingtechnology such as pix2pix so as to construct a design image generationnetwork that generates a design image from an observation image. Byinputting the observation image to the design image generation network,a design image is obtained.

In general, the generalization ability and learning speed are expectedto improve by eliminating the bias contained in the learning data. Thus,an observation image and a design image that have been subjected topreprocesses such as normalization, standardization, decorrelation, andwhitening may be used in learning.

The operator may designate the examination subject S and allow learningso that a learned network is generated immediately before start of theexamination.

An existing matching algorithm is used for aligning the design imagewith the reference image. For example, a kernel of a predetermined size(m pixels×n pixels) is moved and rotated relative to the observationimage or the reference image to calculate the sum of absolute difference(SAD), which is the sum of the absolute differences of pixel values, andthe observation image is aligned on the basis of the position where theSAD is the smallest. When the kernel needs to be expanded or contracted,the kernel size is adjusted on the basis of the size of the observationimage obtained from the magnification of the optical system of themicroscope device 20 and the actual dimension information of the designimage.

Generation and alignment of the design image may be performed by othermethods. For example, an edge image that serves as a design image may begenerated from the observation image by performing edge detection on theobservation image, and the edge image may be aligned with the referenceimage. A traditional filter process, deep-learning edge assumption(Ruohui Wang, Edge Detection Using Convolutional Neural Network,Advances in Neural Network—ISNN 2016, 13th International Symposium onNeural Network, pp. 12-20, 2016), or the like is used for the edgedetection.

As illustrated in FIG. 3A, the navigation map generating unit 8generates from the reference image a reference image A for navigation.For example, the reference image A for navigation is a copy of thereference image. Hereinafter, the reference image A for navigation isreferred to as a copy reference image. In the example illustrated inFIG. 3A, the copy reference image A includes two areas S1 and S2 whereelectronic circuits are formed.

Next, on the basis of the alignment result of the design image relativeto the reference image conducted by the alignment unit 7, the navigationmap generating unit 8 records, on the copy reference image A, theposition of the area that corresponds to the area in the reference imagewith which the design image has been aligned. Recording of the positionis carried out by, for example, placing the design image in the areawhich is within the copy reference image A and which corresponds to thearea in the reference image with which the design image has beenaligned, and then registering the design image. The design image may beregistered by blacking out the corresponding area.

The alignment unit 7 and the navigation map generating unit 8 repeataligning of the design image and recording of the position on the copyreference image A until the alignment unit 7 and the navigation mapgenerating unit 8 receive an examination completion notification. As aresult, as illustrated in FIG. 3B, the navigation map generating unit 8generates a navigation map B, on which the positions of the observationareas already observed by the operator are recorded, from the copyreference image A. In FIG. 3B, rectangles indicated by broken linesindicate areas where the positions of the observation images or thedesign images have been recorded, and a rectangle with hatchingindicates an area D where the position of the observation image or thedesign image is recorded last. For the sake of simplicity of thedrawing, illustration of detailed structures in the areas S1 and S2 isomitted from the drawings in FIG. 3B and onward. The examinationcompletion notification is input to the processor unit 6 on the basis ofthe input by the operator to the microscopic examination device 1. Forexample, when the visual examination of the examination subject S iscompleted, the operator operates the operation unit installed in themicroscopic examination device 1 to send the examination completionnotification from the operation unit to the processor unit 6.

Upon receiving the examination completion notification, the navigationmap generating unit 8 ends generation of the navigation map B. Whenthere is an unobserved area that has not been observed by the operatorwithin the examination subject S, the generated navigation map Bincludes the unobserved area where the position of the design image hasnot been recorded.

As illustrated in FIG. 4A, the access calculation unit 9 detects anunobserved area E from the navigation map B. Since the observation areais manually moved by the operator, the size of the unobserved areavaries. The access calculation unit 9 may detect, as one unobserved areaE, an area having the same size as the observation area, which is thevisual field of the objective lens 22. In FIG. 4A, only one of multipleunobserved areas present in the navigation map B is indicated by asolid-line rectangle.

Next, on the basis of the relative position in the navigation map Bbetween the current observation area D and the unobserved area E, theaccess calculation unit 9 calculates the direction and the amount ofmovement from the current observation area D to the unobserved area E.For example, the access calculation unit 9 uses, as the currentobservation area D, an area of the design image last recorded on thenavigation map B by the navigation map generating unit 8. The directionand amount of movement are calculated, for example, as a directionalvector on the basis of the number of pixels between the center of thecurrent observation area D and the center of the unobserved area E onthe navigation map B.

On the basis of the direction and amount of movement, for example, thedirectional vector, calculated by the access calculation unit 9, thenavigation presenting unit 10 displays on the navigation map B an accessmethod C to the unobserved area E so as to present the access method Cto the operator. FIGS. 4A and 4B illustrate examples of how the accessmethod C is displayed. In FIG. 4A, the display of the access method Cinvolves an arrow that indicates the directional vector from the currentobservation area D to the unobserved area E. In other words, the tail ofthe arrow indicates the position of the current observation area D, andthe head of the arrow indicates the position of the unobserved area E.In FIG. 4B, the display of the access method C involves two arrows thatrespectively indicate the X component and the Y component of thedirectional vector.

The display of the access method C may be an animation of the handle 23a that indicates the direction and amount of rotation of the handle 23 ain the X direction and the Y direction of the stage 21.

The navigation map B on which the access method C is displayed is outputfrom the navigation presenting unit 10 to the display device 30, and isdisplayed on the display device 30. The navigation map B that is beinggenerated may be displayed on the display device 30, and the displayednavigation map B may be updated sequentially.

Next, a navigation method that guides the observation area of theexamination subject S of the microscope device 20 to an area not yetobserved is described with reference to FIGS. 5A to 5C. This navigationmethod involves an operation of the microscopic examination device 1.

The operator repeats observation of the observation area of theexamination subject S through the ocular lens 24 and moving of theobservation area so as to visually examine all parts of the examinationsubject S.

In parallel with the visual examination by the operator, the observationarea is imaged and the navigation map B is generated by the microscopicexamination device 1.

Specifically, as illustrated in FIG. 5A, a reference image is acquiredby the reference image input unit 5 (step S1), and a copy referenceimage A for generating a navigation map is generated from the referenceimage (step S2). An observation image, which is the image of theobservation area that the operator is observing, is acquired by theimaging unit 2 (step S3).

The reference image is input from the reference image input unit 5 tothe processor unit 6, and the observation image is input from theimaging unit 2 to the processor unit 6.

Next, the alignment unit 7 aligns the observation image with thecorresponding area in the reference image (step S4). Specifically, asillustrated in FIG. 5B, the reference image and the observation imageare read (steps S41 and S42), and a design image, which is anobservation image for alignment, is generated from the observation image(step S43). Next, the similarity between the reference image and thedesign image is calculated (step S44), and the area having the highestsimilarity within the reference image is aligned with the design image(step S45).

Next, a navigation map B is generated by the navigation map generatingunit 8 from the reference image (step S5). Specifically, as illustratedin FIG. 5C, a copy reference image A, which is a reference image fornavigation, is generated from the reference image (step S51). Next, onthe basis of the alignment result in step S4, the position of the designimage is recorded on the copy reference image A (step S52).

The steps S3 to S5 are repeated until the visual examination of theexamination subject S by the operator is completed, and as a result,positions of the design images are added to the copy reference image Aand the navigation map B is generated.

After completion of the visual examination by the operator (YES in stepS6), the access calculation unit 9 detects an unobserved area E in thenavigation map B, and calculates the direction and amount of movementfrom the current observation area D to the unobserved area E (step S7).Next, the navigation presenting unit 10 displays, on the navigation mapB, the access method C indicating the direction and amount of movement(step S8), and the navigation map B on which the access method C isdisplayed is displayed on the display device 30.

The operator can easily identify the already observed areas of theexamination subject S through the navigation map B displayed on thedisplay device 30. In addition, on the basis of the access methoddisplayed on the navigation map B, the operator can easily recognizewhether an unobserved area E is present and, if so, how to access thecurrent observation area D to the unobserved area E. Particularly, byusing, as a display of the access method C, a graphic indicating thedirection and amount of movement to the unobserved area E, for example,an arrow or an animation of the handle 23 a, the operator can moreeasily and intuitively recognize whether an unobserved area E is presentand how to access the unobserved area E.

FIG. 6 illustrates a modification of the present embodiment.

As illustrated in FIG. 6 , a microscopic examination device 101 mayfurther include an access sequence determining unit 11.

As illustrated in FIG. 7A, after the navigation map generating unit 8finishes generation of the navigation map B, when there are more thanone unobserved areas in the navigation map B, the access sequencedetermining unit 11 determines the sequence in which the unobservedareas are to be navigated (step S9).

FIG. 7B illustrates one example of the method for determining the accesssequence. In the example illustrated in FIG. 7B, the access sequencedetermining unit 11 detects unobserved areas from the navigation map B(step S91). Next, in order to calculate the relative position betweenthe current observation area D and each of the unobserved areas, theaccess sequence determining unit 11 calculates the direction and amountof movement from the current observation area D to each of theunobserved areas in the navigation map B (step S92). The method forcalculating the direction and amount of movement are the same as themethod for calculating the direction and amount of movement employed inthe access calculation unit 9. Next, the access sequence determiningunit 11 detects the adjacency state of the unobserved areas (step S93),and, on the basis of the distances from the current observation area Dto each of the unobserved areas and the adjacency state of theunobserved areas, the access sequence determining unit 11 determines theaccess sequence of the unobserved areas (step S94).

FIG. 8 illustrates a specific example of the method for determining theaccess sequence.

In this example, N unobserved areas are assigned with distance ranks(from 1 to N) according to the distance from the current observationarea D. In FIG. 8 , rectangles of solid lines in the navigation map Bindicate the unobserved areas. The distance rank of the unobserved areahaving the shortest distance is 1, and the distance rank of theunobserved area having the longest distance is N.

Next, the adjacency state between the distance rank 1-unobserved areaand other unobserved areas is detected, and access ranks are assigned tothe rank 1-unobserved area and the unobserved areas adjacent thereto.The adjacent unobserved areas are unobserved areas that are present in 8neighbours of a particular unobserved area. Specifically, an access rank1-0 is assigned to the unobserved area having a distance rank of 1, andneighbor ranks serving as access ranks are assigned to the adjacentunobserved areas that are present in 8 neighbors of the rank1-unobserved area. In the example illustrated in FIG. 8 , a rank 1-1 anda rank 1-2 are respectively assigned, as neighbor ranks, to the twoadjacent unobserved areas adjacent to the rank-1 unobserved area. Theneighbor rank is, for example, successively assigned clockwise. Theadjacent unobserved areas assigned with the neighbor ranks are labeledas assigned.

Next, the unobserved areas labeled as assigned are set aside, theadjacency state of the unobserved area having the next-highest distancerank is detected, and access ranks are assigned to the next-highest-rankunobserved area and unobserved areas adjacent thereto. Then the sameprocess is repeated until all unobserved areas are assigned with accessranks. The access ranks constitute the access sequence.

The access sequence determination method described above is merely anexample, and the access sequence may be determined by other methods. Forexample, the access sequence may be determined such that an unobservedarea that can be accessed from the current observation area D by linearmovement, for example, an unobserved area located above or under or onthe right- or left-hand side of the current observation area D, isassigned a higher rank.

The navigation presenting unit 10 displays on the navigation map B anaccess method for accessing two or more unobserved areas according tothe access sequence.

According to this modification, a moving route of the examinationsubject S for efficiently accessing the unobserved areas in order ofproximity from the current observation area D can be presented to theoperator, and the efficiency of examination can be improved.

Second Embodiment

Next, a microscopic examination device and a microscopic examinationsystem according to a second embodiment of the present invention aredescribed with reference to the drawings.

The microscopic examination device and the microscopic examinationsystem according to this embodiment have identical device structures tothe microscopic examination device 101 and the microscopic examinationsystem 102 illustrated in FIG. 6 , but are different from themicroscopic examination device 101 and microscopic examination system102 in the process conducted in the navigation presenting unit 10.

As illustrated in FIG. 9 , while the observation area is being moved toan unobserved area E, the navigation presenting unit 10 presents on anavigation map B the access speed that changes according to the distancefrom the current observation area to the unobserved area E in additionto the access method C from the current observation area to theunobserved area E (step S11).

Thus, as the observation area is being moved, the position of theobservation area and the distance from the observation area to theunobserved area E are detected and monitored (step S10).

The position of the moving observation area is detected by the alignmentunit (position detecting unit) 7. In other words, as the observationarea is being moved, the imaging unit 2 acquires an observation image ata particular timing. The alignment unit 7 aligns the observation imagewith the reference image, and detects, as the position of the currentobservation area, the position of the observation image in the referenceimage.

The distance from the position of the observation area detected by thealignment unit 7 to the unobserved area E is detected by the accesscalculation unit 9. The access calculation unit 9 calculates thedistance from the current observation area to the unobserved area E bythe same method as the method for calculating the amount of movementfrom the current observation area D to the unobserved area E describedin the first embodiment.

The navigation presenting unit 10 displays, on the navigation map B, aspeed indicator suggesting the access speed on the basis of the distancefrom the observation area to the unobserved area E. When the distancefrom the observation area to the unobserved area E is short, the speedindicator suggests slowing down the access speed. For example, asillustrated in FIGS. 10A and 10B, the arrow C indicating the directionand amount of movement is a speed indicator, and the thickness of thearrow C indicates the access speed. In such a case, the shorter thedistance from the observation area to the unobserved area E, the thinnerthe arrow C. Alternatively, as illustrated in FIGS. 11A and 11B, theaccess speed may be indicated by the number of “<” signs in the arrow Cindicating the direction and amount of movement. In such a case, theslower the suggested access speed, the fewer the signs.

According to this embodiment, the following effects are exhibited inaddition to the effects of the first embodiment. Specifically, since thespeed indicator on the navigation map B changes according to the changesin the distance from the observation area to the unobserved area E whileaccessing the unobserved area E, the operator can easily and intuitivelyrecognize the appropriate access speed from the speed indicator.

The moving speed of the examination subject S can be appropriatelyadjusted. For example, when the observation area is far from theunobserved area E, the examination subject S is moved quickly relativeto the objective lens 22, and when the observation area is near theunobserved area E, the examination subject S is slowly moved relative tothe objective lens 22.

The navigation presenting unit 10 may be equipped with a haptic deviceinstalled in the handle 23 a, and the haptic device may present, to thehand of the operator maneuvering the handle 23 a, a haptic sensecorresponding to the access speed.

For example, the haptic device may generate a reaction force against themaneuvering of the handle 23 a by the operator, and this reaction forcemay be decreased with the increasing access speed.

Third Embodiment

Next, a microscopic examination device 103 and a microscopic examinationsystem 300 according to a third embodiment of the present invention aredescribed with reference to the drawings.

The microscopic examination device 103 according to this embodiment isapplied to a microscope device 20 equipped with a stage 21. Asillustrated in FIG. 12 , the microscopic examination device 103 furtherincludes an encoder 12 that is installed in the microscope device 20 anddetects the position of the stage 21. Other structures of themicroscopic examination device 103 and the microscopic examinationsystem 300 are identical to those of the microscopic examination device101 and the microscopic examination system 102 illustrated in FIG. 6 .

The encoder 12 detects the position of the stage 21 during and after thevisual examination. The position of the stage 21 detected by the encoder12 is the relative position relative to the examination startingposition.

As illustrated in FIG. 13A, before starting the visual examination, theoperator moves the observation area to the examination startingposition, and commands the imaging unit 2 to acquire an observationimage at the examination starting position (step S12). The examinationstarting position is a predetermined position within the examinationsubject S.

Since the operator moves the observation area by hand, it is possiblethat the position of the observation area would deviate from theexamination starting position. The examination starting position may beautomatically corrected to the position of the observation area set bythe operator. For example, the position of the observation area in thereference image may be detected by aligning the observation image at theexamination starting position with the reference image, and theexamination starting position may be changed to the detected position.

Next, the setting of the encoder 12 is initialized (step S13). Then thevisual examination is started, and the imaging unit 2 acquires anobservation image on the basis of the movement of the observation area(step S3).

In step S4, the alignment unit 7 aligns the observation image with thereference image on the basis of the position of the stage 21 detected bythe encoder 12.

Specifically, as illustrated in FIG. 13B, the alignment unit 7calculates the position of the design image in the reference image onthe basis of the examination starting position and the position of thestage 21 detected by the encoder 12 (step S46), and aligns the designimage with the calculated position (step S45).

There may be cases where the design image is rotated with respect to thereference image. After the design image is temporarily aligned, thealignment unit 7 may rotate the design image with respect to thereference image and calculate the similarity between the reference imageand the design image at different rotation angles (step S44), and mayfinally align the design image with the reference image at a rotationangle at which the highest similarity is detected (step S45).

In step S10 also, the alignment unit 7 detects the position of themoving observation area on the basis of the examination startingposition and the position of the stage 21 detected by the encoder 12.

According to this embodiment, by using the position of the stage 21detected by the encoder 12, the time taken to search for the area thatcorresponds to the observation image in the reference image can beshortened. Moreover, the accuracy of aligning the observation image withthe reference image and the accuracy of detecting the position of themoving observation area are improved, and thus the navigation accuracyto the unobserved area E can be improved. Thus, overlooking of theunobserved area E caused by alignment errors or the like can be avoided.

In the respective embodiments described above, the access calculationunit 9 calculates the direction and amount of movement to the unobservedarea E; alternatively, the access calculation unit 9 may calculate onlythe direction of movement to the unobserved area E. In such a case, thenavigation presenting unit 10 presents the access method solely on thebasis of the direction of movement. For example, the navigationpresenting unit 10 displays, on the navigation map B, an arrow that hasa desired length and is directed from the current observation area D tothe unobserved area E. The operator can reach the unobserved area E bymoving the observation area in the direction indicated by the arrow.

As a result, the above-described embodiments lead to the followingaspects.

An aspect of the present invention is directed to a microscopicexamination device that includes: an imaging unit that images anobservation area of an examination subject observed with a microscopedevice so as to acquire an image of the observation area; an alignmentunit that aligns the image of the observation area with an area within areference image, the area corresponding to the image of the observationarea, the reference image including an entirety of the examinationsubject; a navigation map generating unit that generates a navigationmap from the reference image by recording, on the reference image, aposition of the image of the observation area in the reference image; anaccess calculation unit that calculates a direction of movement to anunobserved area in the navigation map, the unobserved area being an areawhere the position of the image of the observation area is not recorded;and a navigation presenting unit that presents an access method to theunobserved area on the basis of the direction of movement.

According to this aspect, in parallel with the visual examination of theexamination subject by using a microscope device operated by theoperator, an image of the observation area of the examination subject isacquired and a navigation map is generated by the microscopicexamination device. That is, the imaging unit acquires the image of theobservation area, and the alignment unit aligns the image of theobservation area with the reference image. Then the navigation mapgenerating unit records, on the reference image, the position of theimage of the observation area.

The operator manually moves the examination subject or the stage of themicroscope device on which the examination subject is placed to move theobservation area, and visually examines all parts of the examinationsubject. As the observation area is moved, the imaging unit acquiresimages of observation areas at different positions in the examinationsubject, and the positions of the observation areas are added to thereference image. As a result, the positions of the areas observed by theoperator are sequentially recorded on the reference image, and anavigation map is generated from the reference image. When an unobservedarea not yet observed by the operator remains in the examinationsubject, the navigation map includes the unobserved area where theposition of the image of the observation area has not been recorded.Next, the access calculation unit calculates the direction of movementto the unobserved area in the navigation map, and the access method tothe unobserved area is presented by the navigation presenting unit.

Using the navigation map, the operator can easily recognize whether anunobserved area is present in the examination subject. When anunobserved area is present, the operator, on the basis of the accessmethod presented by the navigation presenting unit, can easily recognizethe method for moving the observation area of the microscope device tothe unobserved area. In addition, by using the alignment of thereference image and the image of the observation area, the position ofthe observation area to be manually moved can be detected.

In the aspect described above, the access calculation unit may furthercalculate an amount of movement to the unobserved area, and thenavigation presenting unit may present the access method on the basis ofthe direction and amount of movement.

According to such a structure, a more specific access method to theunobserved area can be presented.

In the aspect described above, the microscopic examination device mayfurther include an encoder that is installed in the microscope deviceand detects a position of a stage of the microscope device, and, on thebasis of the position of the stage detected by the encoder, thealignment unit may calculate a position of the area in the referenceimage, the area corresponding to the image of the observation area.

According to such a structure, the position in the reference image to bealigned with the image of the observation area can be calculated withless computation efforts, and thus the time taken to align the image ofthe observation area with the reference image can be shortened.Moreover, the alignment accuracy of the image of the observation areawith the reference image can be improved.

In the aspect described above, the microscopic examination device mayfurther include an access sequence determining unit that determines anaccess sequence of two or more unobserved areas, and the navigationpresenting unit may present the access method for accessing the two ormore unobserved areas according to the access sequence.

According to such a structure, when two or more unobserved areas arepresent on the navigation map, the order in which the two or moreunobserved area are accessed can be presented to the operator.

In the aspect described above, the microscopic examination device mayfurther include a position detecting unit that detects the position ofthe observation area moving toward the unobserved area, the navigationpresenting unit may present an access speed on the basis of the positionof the observation area detected by the position detecting unit, and theaccess speed may change according to a distance from the detectedposition of the observation area to the unobserved area.

The access speed is the speed of moving the observation area to theunobserved area, that is, the speed of moving the examination subject orthe stage. According to such a structure, an appropriate access speedfor the distance to the unobserved area can be presented to theoperator.

In the aspect described above, the navigation presenting unit maydisplay, on the navigation map, an indicator indicating the accessspeed. Alternatively, the navigation presenting unit may include ahaptic device installed in a handle used to manually maneuver a stage ofthe microscope device, and the haptic device may present, to the hand ofthe operator maneuvering the handle, a haptic sense corresponding to theaccess speed.

According to such a structure, the access speed is visually orhaptically presented to the operator. Thus, the operator can moreintuitively recognize the access speed.

Another aspect of the present invention is directed to a navigationmethod for guiding an observation area of an examination subject to beobserved with a microscope device, the navigation method including:imaging the observation area to acquire an image of the observationarea; aligning the image of the observation area with an area in areference image, the area corresponding to the image of the observationarea, the reference image including an entirety of the examinationsubject; generating a navigation map from the reference image byrecording, on the reference image, a position of the image of theobservation area in the reference image; calculating a direction ofmovement to an unobserved area in the navigation map, the unobservedarea being an area where the position of the image of the observationarea is not recorded; and presenting an access method to the unobservedarea on the basis of the direction of movement.

The present invention affords the advantage that, in manually operatedmicroscopic examination, the operator can more easily recognize a methodfor accessing an unobserved area of the examination subject.

REFERENCE SIGNS LIST

-   -   1, 101, 103 microscopic examination device    -   2 imaging unit    -   3 imaging control unit    -   4 memory unit    -   5 reference image input unit    -   6 processor unit    -   7 alignment unit, position detecting unit    -   8 navigation map generating unit    -   9 access calculation unit    -   10 navigation presenting unit    -   11 access sequence determining unit    -   12 encoder    -   20 microscope device    -   21 stage    -   30 display device    -   100, 102, 300 microscopic examination system    -   B navigation map    -   S examination subject

The invention claimed is:
 1. A microscopic examination devicecomprising: a camera that images an observation area of an examinationsubject observed with a microscope device so as to acquire an image ofthe observation area; and one or more processors comprising hardware,the one or more processors being configured to: align the image of theobservation area with an area in a reference image, the areacorresponding to the image of the observation area, the reference imageincluding an entirety of the examination subject; generate a navigationmap from the reference image by recording, on the reference image, aposition of the image of the observation area in the reference image;calculate a direction of movement to an unobserved area in thenavigation map, the unobserved area being an area where the position ofthe image of the observation area is not recorded; and present an accessmethod to the unobserved area on a basis of the direction of movement.2. The microscopic examination device according to claim 1, wherein: thecalculating of the direction of movement further calculates an amount ofmovement to the unobserved area, and the presenting of the access methodpresents the access method on a basis of the direction and amount ofmovement.
 3. The microscopic examination device according to claim 1,further comprising: an encoder that is installed in the microscopedevice and detects a position of a stage of the microscope device,wherein, on a basis of the position of the stage detected by theencoder, the aligning of the image calculates a position of the area inthe reference image, the area corresponding to the image of theobservation area.
 4. The microscopic examination device according toclaim 1, wherein the one or more processors are further configured todetermine an access sequence of two or more unobserved areas, andwherein the presenting of the access method presents the access methodfor accessing the two or more unobserved areas according to the accesssequence.
 5. The microscopic examination device according to claim 1,wherein the one or more processors are further configured to detect theposition of the observation area moving toward the unobserved area, andwherein the presenting of the access method presents an access speed ona basis of the position of the detected observation area, and the accessspeed changes according to a distance from the detected position of theobservation area to the unobserved area.
 6. The microscopic examinationdevice according to claim 5, wherein the presenting of the access methoddisplays, on the navigation map, an indicator indicating the accessspeed.
 7. The microscopic examination device according to claim 5,further comprising a haptic device installed in a handle used tomanually maneuver a stage of the microscope device, and the presentingof the access method presents, to a hand of an operator maneuvering thehandle, a haptic sense corresponding to the access speed via the hapticdevice.
 8. A navigation method for guiding an observation area of anexamination subject to be observed with a microscope device, thenavigation method comprising: imaging the observation area to acquire animage of the observation area; aligning the image of the observationarea with an area in a reference image, the area corresponding to theimage of the observation area, the reference image including an entiretyof the examination subject; generating a navigation map from thereference image by recording, on the reference image, a position of theimage of the observation area in the reference image; calculating adirection of movement to an unobserved area in the navigation map, theunobserved area being an area where the position of the image of theobservation area is not recorded; and presenting an access method to theunobserved area on a basis of the direction of movement.